Skip to content

otel: add test for document equivalence between agent filestream input and filebeat receiver#6681

Merged
mauri870 merged 22 commits intoelastic:mainfrom
mauri870:hybrid-agent-test
Feb 14, 2025
Merged

otel: add test for document equivalence between agent filestream input and filebeat receiver#6681
mauri870 merged 22 commits intoelastic:mainfrom
mauri870:hybrid-agent-test

Conversation

@mauri870
Copy link
Member

@mauri870 mauri870 commented Jan 31, 2025 •
edited
Loading

What does this PR do?

This PR adds a test that compares a log line ingested by a filestream agent input and a filebeat OTel receiver with equivalent configuration.

Checklist

  • I have read and understood the pull request guidelines of this project.
  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool
  • I have added an integration test or an E2E test

Related issues

@mauri870 mauri870 added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team Cleanup skip-changelog backport-8.x Automated backport to the 8.x branch with mergify labels Jan 31, 2025
@mauri870 mauri870 self-assigned this Jan 31, 2025
@mauri870 mauri870 changed the title Hybrid agent test otel: add test for document equivalence between filebeat and filebeat receiver Jan 31, 2025
@mauri870 mauri870 changed the title otel: add test for document equivalence between filebeat and filebeat receiver otel: add test for document equivalence between agent filestream input and filebeat receiver Jan 31, 2025
@mauri870 mauri870 marked this pull request as ready for review January 31, 2025 14:03
@mauri870 mauri870 requested a review from a team as a code owner January 31, 2025 14:03
@elasticmachine
Copy link
Contributor

elasticmachine commented Jan 31, 2025

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@mauri870 mauri870 requested review from leehinman and removed request for kaanyalti January 31, 2025 14:03
mauri870
mauri870 commented Jan 31, 2025
View reviewed changes
leehinman
leehinman reviewed Jan 31, 2025
View reviewed changes
Copy link
Contributor

@leehinman leehinman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think if you add the default processors listed at https://github.com/elastic/beats/blob/76f40863c9918ee1a691a2bb29cbab98a5d83878/x-pack/filebeat/cmd/root.go#L48 and add the processors listed in the generate.go below you should get a much shorter list of fields to ignore.

@mauri870 mauri870 marked this pull request as draft February 3, 2025 12:33
@mauri870 mauri870 marked this pull request as ready for review February 6, 2025 14:16
khushijain21
khushijain21 previously approved these changes Feb 7, 2025
View reviewed changes
@khushijain21 khushijain21 dismissed their stale review February 7, 2025 06:43

tests are failing

@pierrehilbert
Copy link
Contributor

pierrehilbert commented Feb 7, 2025

I don't think the failing tests are related to this PR:

=== RUN   TestDownloadFile
    utils_test.go:21: 
        	Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-aws-1738851441003538166/elastic/elastic-agent/dev-tools/mage/downloads/utils_test.go:21
        	Error:      	Expected nil, but got: &fmt.wrapError{msg:"downloading file https://www.elastic.co/robots.txt: GET request failed with 504", err:(*errors.errorString)(0xc00085a970)}
        	Test:       	TestDownloadFile
--- FAIL: TestDownloadFile (20.57s)

Will update the branch to have them running again and hope the gateway failure was temporary

@mergify
Copy link
Contributor

mergify bot commented Feb 10, 2025

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b hybrid-agent-test upstream/hybrid-agent-test
git merge upstream/main
git push upstream hybrid-agent-test

leehinman
leehinman approved these changes Feb 10, 2025
View reviewed changes
Copy link
Contributor

@leehinman leehinman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@elastic-sonarqube
Copy link

elastic-sonarqube bot commented Feb 11, 2025

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

@mauri870 mauri870 merged commit 41882ac into elastic:main Feb 14, 2025
14 checks passed
mergify bot pushed a commit that referenced this pull request Feb 14, 2025
@mauri870 @mergify
otel: add test for document equivalence between agent filestream inpu…
c846821 
…t and filebeat receiver (#6681)

* otel: add test for agent hybrid

* add missing fingerprint config options to index small files

* ensure new lines are written to the input file

* disable compression level so we inspect the requests

* add queue timeout to normal beats and separate path.home for fbreceiver

* add path.home for normal filebeat

* fix agent not starting

* print elastic-agent output if test fails

* add note for metadata field

* fail if ignored field is not present in both maps

* add host metadata processor, remove 'host.*' from ignored fields

* Use global processor

* more strict rules for ignored fields to avoid false positives

* use add_fields to add missing fields, use a single index

* add add_cloud_metadata processor

* avoid checking cmd.Wait error since it is bogus

(cherry picked from commit 41882ac)
mergify bot pushed a commit that referenced this pull request Feb 14, 2025
@mauri870 @mergify
otel: add test for document equivalence between agent filestream inpu…
bb70746 
…t and filebeat receiver (#6681)

* otel: add test for agent hybrid

* add missing fingerprint config options to index small files

* ensure new lines are written to the input file

* disable compression level so we inspect the requests

* add queue timeout to normal beats and separate path.home for fbreceiver

* add path.home for normal filebeat

* fix agent not starting

* print elastic-agent output if test fails

* add note for metadata field

* fail if ignored field is not present in both maps

* add host metadata processor, remove 'host.*' from ignored fields

* Use global processor

* more strict rules for ignored fields to avoid false positives

* use add_fields to add missing fields, use a single index

* add add_cloud_metadata processor

* avoid checking cmd.Wait error since it is bogus

(cherry picked from commit 41882ac)
This was referenced Feb 14, 2025
Merged
Merged
mauri870 added a commit that referenced this pull request Feb 14, 2025
@mergify @mauri870
otel: add test for document equivalence between agent filestream inpu…
2161adb 
…t and filebeat receiver (#6681) (#6868)

* otel: add test for agent hybrid

* add missing fingerprint config options to index small files

* ensure new lines are written to the input file

* disable compression level so we inspect the requests

* add queue timeout to normal beats and separate path.home for fbreceiver

* add path.home for normal filebeat

* fix agent not starting

* print elastic-agent output if test fails

* add note for metadata field

* fail if ignored field is not present in both maps

* add host metadata processor, remove 'host.*' from ignored fields

* Use global processor

* more strict rules for ignored fields to avoid false positives

* use add_fields to add missing fields, use a single index

* add add_cloud_metadata processor

* avoid checking cmd.Wait error since it is bogus

(cherry picked from commit 41882ac)

Co-authored-by: Mauri de Souza Meneguzzo <mauri870@gmail.com>
mauri870 added a commit that referenced this pull request Feb 19, 2025
@mergify @mauri870
otel: add test for document equivalence between agent filestream inpu…
003c923 
…t and filebeat receiver (#6681) (#6867)

* otel: add test for agent hybrid

* add missing fingerprint config options to index small files

* ensure new lines are written to the input file

* disable compression level so we inspect the requests

* add queue timeout to normal beats and separate path.home for fbreceiver

* add path.home for normal filebeat

* fix agent not starting

* print elastic-agent output if test fails

* add note for metadata field

* fail if ignored field is not present in both maps

* add host metadata processor, remove 'host.*' from ignored fields

* Use global processor

* more strict rules for ignored fields to avoid false positives

* use add_fields to add missing fields, use a single index

* add add_cloud_metadata processor

* avoid checking cmd.Wait error since it is bogus

(cherry picked from commit 41882ac)

Co-authored-by: Mauri de Souza Meneguzzo <mauri870@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@leehinman leehinman leehinman approved these changes

@khushijain21 khushijain21 khushijain21 approved these changes

@blakerouse blakerouse Awaiting requested review from blakerouse blakerouse is a code owner automatically assigned from elastic/elastic-agent-control-plane

Assignees

@mauri870 mauri870

Labels

backport-8.x Automated backport to the 8.x branch with mergify backport-9.0 Automated backport to the 9.0 branch Cleanup skip-changelog Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@mauri870 @elasticmachine @pierrehilbert @leehinman @khushijain21